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The RICIS Concept 


The University of Houston-Clear Lake established the Research Institute for 
Computing and Information Systems (RICIS) in 1986 to encourage the NASA 
Johnson Space Center (JSC) and local industry to actively support research 
In the computing and information sciences. As part of this endeavor, UHCL 
proposed a partnership with JSC to jointly define and manage an integrated 
program of research in advanced data processing technology needed for JSC’s 
main missions, including administrative, engineering and science responsi- 
bilities. JSC agreed and entered into a continuing cooperative agreement 
with UHCL beginning in May 1986, to jointly plan and execute such research 
through RICIS. Additionally, under Cooperative Agreement NCC 9-16, 
computing and educational facilities are shared by the two institutions to 
conduct the research. 

The UHCL/RICJS mission is to conduct, coord in ate, and disseminate research 
and professional level education in computing and Information systems to 
serve the needs of the government, industry, community and academia. 
RICIS combines resources of UHCL and its gateway affiliates to research and 
develop materials, prototypes and publications on topics of mutual interest 
to its sponsors and researchers. Within UHCL, the mission is being 
implemented through interdisciplinary involvement of faculty and students 
from each of the four schools: Business and Public Administration, Educa- 
tion, Human Sciences and Humanities, and Natural and Applied Sciences. 
RICIS also collaborates with industiy in a companion program. This program 
is focused on serving the research and advanced development needs of 
industry. 

Moreover, UHCL established relationships with other universities and re- 
search organizations, having common research interests, to provide addi- 
tional sources of expertise to conduct needed research. For example, UHCL 
has entered into a special partnership with Texas A&M University to help 
oversee RICIS research ani education programs, while other research 
organizations are involved via the “gateway" concept 

A major role of RICIS then is to find the best match of sponsors, researchers 
and research objectives to advance knowledge in the computing and informa- 
tion sciences. RICIS, working jointly with its sponsors, advises on research 
needs, recommends principals for conducting the research, provides tech- 
nical and administrative support to coordinate the research and integrates 
technical results Into the goals of UHCL, NASA/ JSC and industry. 
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EXECUTIVE SUMMARY 


Measuring an organization’s performance is an essential component in the recent drive 
toward quality in U.S. business and government operations. Managers can no longer rely 
exclusively on their intuition or "feel"; they must use real data to guide their organizations 
toward their goals. The movement toward "management by results" is affecting all 
institutions— business , government, and education. 

The Information Services Directorate (ISD) of the NASA Johnson Space Center (JSC) 
requested a study of this movement among information system (IS) organizations in industry. 
ISD was establishing a new support contract and reorganizing itself to better serve its 
customers. They wanted to develop their performance measurement in light of industry’s 
experience. 

Researchers at the University of Houston-Clear Lake interviewed IS executives at six 
large corporations in finance, oil and gas, utilities, and waste disposal. The executives 
described their IS measurement system, how it had developed, and how satisfied they were 
with it. They also discussed what they were doing to improve their systems and what they 
thought NASA should consider as they embarked on their own system. The respondents 
were extraordinarily generous with their time and shared a number of valuable insights with 
the interviewers. 

The most important insight was that establishing a performance measurement system 
was more an organizational than a technical issue. Securing the commitment of employees 
and executives to collect and use measurement data was the most crucial element. That 
commitment in turn created new ways of thinking and acting that drove the organization to 
improve. As one executive put it, "The things you measure improve; the things you don’t 
measure, don’t. " Performance measurement required a cultural change to be successful and 
touched off similar cultural changes of its own. 

Another important insight was that the customer is central to measuring performance. 
Firms measured their customer satisfaction in various ways. Some routinely surveyed their 
customer; others met with customer executives to discuss next year’s strategic plans. 

However they did it, the customer’s view was paramount, particularly in those organizations 
which had been conducting formal measurement the longest. 

Finally, respondents emphasized how long a suitable measurement took to develop. 
One firm negotiated with their outsourcing contractor for over two years before they had a 
measurement system that both could support. Other firms had been measuring their 
performance for ten years and were still modifying and improving it. Firms hired 
consultants and research services to help compare themselves with their industry; they 
devoted internal resources to preparing reports and analyzing changes. Performance 
measurement is not free; rather it is an investment which firms are making in their long-term 
improvement. 
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Respondents also shared their experience in four specific domains of performance 
measurement: customer satisfaction, data processing, application development and end-user 
computing. Firms used two primary strategies for gauging the satisfaction of their 
customers: end-user surveys and executive interviews. These data were used as the final 
measure of the organization’s performance and the success of its improvement strategies. 

Two firms in the sample did not have customer satisfaction measures. One had 
recently discontinued their customer survey under a cost-cutting and downsizing exercise. 

One was currently reluctant to institute customer surveys for fear of raising customer 
expectations beyond what the organization could deliver. 

The organizations which had such customer satisfaction measures did in fact report an 
initial period when customers demanded extraordinary service. Responding in good faith, 
however, the IS organization soon impressed customers with their willingness to change and 
their desire to do the best job possible. The result from successfully using customer 
satisfaction measures was a partnership between IS and its customers that resulted in 
significant improvement in service and joint decisions about future improvements. 

Five firms managed large data processing centers. Their measurement systems were 
stable and mature, resulting from ten years of development. Most firms distributed monthly 
hardcopy reports of numerous processing indicators. The most important measures were 
availability and response time. The reports were distributed widely to both IS and customer 
organizations. One firm maintained an on-line system that presented a daily report-card to 
its major data processing centers. 

Few of these measures, however, contained preset criteria or targets to be met. The 
on-line system did require specific availability and response time and subtracted points from 
the center’s overall score when values fell below those targets. Another firm had negotiated 
a detailed set of metrics with its major support contractor, requiring a specific level of 
service for every measure they took. 

Firms were expending most of their effort to improve performance measurement in 
application development. Previous systems were largely time and effort reporting by project. 
Productivity and quality statistics were calculated using source lines of code (SLOC) as the 
unit of work. Firms were now recalibrating their measures in terms of function points, a 
presumably more valid measure of work in software development. They were measuring the 
function points in their existing and new systems as the basis for re-defined productivity and 
quality measures. 

Firms were also linking their measures to critical business issues in application 
development. Previous systems merely reported the time and effort against projects, but 
other measures like quality and cycle-time were difficult to capture. They are now 
developing measures explicitly for the most important issues in their development areas. 
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In the absence of clear and valid measures for application development, firms 
executed service agreements with their customers detailing the specific measures to be used 
on each project. One firm also contracted with outside reviewers to conduct a quality audit 
of every major development project every quarter to assess progress to date and risk of 
future problems. 

Finally, the area of end-user computing presented the greatest challenge to 
performance measurement in these firms. The tools and techniques to measure performance 
in end-user computing have not caught up with the new technology and proliferation of 
workstations and local-area networks. Firms have had to settle for measuring their response 
to service requests and troubleshooting as the only valid measure in this area. Some are 
expecting full end-to-end network and workstation measurement tools to be available in two 
to three years. 

Respondents were also asked to share their experience with NASA who was 
developing a new performance measurement system. They described that good working 
relationships among all the people affected by the measurement system were essential to its 
success. They stressed that the development phase should be carefully planned and that 
outside assistance was important. Even then the development could be long, and the results 
would still need further refinement as the system was put in place. Finally, they pointed out 
that cultural change was both a pre-requisite and the result of a successful performance 
measurement system. 
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PERFORMANCE MEASUREMENT FOR INFORMATION SYSTEMS: 

INDUSTRY PERSPECTIVES 

INTRODUCTION 


Performance measurement has become a focal topic for information systems (IS) 
organizations. Historically, IS performance measures have dealt with the efficiency of the 
data processing function. Today, the function of most IS organizations goes beyond simple 
data processing. To understand how IS organizations have developed meaningful 
performance measures that reflect their objectives and activities, a study was undertaken to 
survey industry perspectives on IS performance measurement. This is the final report on that 
study. 

Six IS organizations were interviewed to glean an understanding of their approach to 
performance measurement and how their measurement system was developed. This report 
contains the results of those interviews. The report provides examples of how some of the 
most forward looking companies are shaping their IS processes through measurement. 

The organization of this report provides for 1) background information on this research 
study including the objectives, scope and approach the study follows, 2) description of the 
sample population, 3) trends in performance measurement systems, 4) patterns by 
product/service category, 5) recommendations to NASA as they undertake to reshape their IS 
performance measurement system; and finally, 6) suggestions for managers undertaking the 
development of a meaningful performance measurement system. Included in the appendices 
of this report are beginning thoughts on the presence of a life-cycle to performance measures 
development and a suggested taxonomy for performance measurements. 

All of the firms interviewed were exceedingly generous in their time and information. 
Their generosity also indicated their interest in learning more about industry practice in this 
area. The study team hopes that this report furthers that learning process. 


BACKGROUND 

U.S. industry is responding to the competitive challenge of the global economy by 
adopting the principles of quality management in every line of business. The principles 
involve a renewed customer orientation, an increased attention to process, and a commitment 
to reduce defects, cycle time and cost. 

One component of a quality approach is the measurement of process and outcome 
variables on a regular basis. Managers are adopting the popular phrase, "If you don’t 



measure it, you can’t manage it." The approach is "management by fact" as opposed to 
management by opinion, intuition, or political advantage. Few people would feel 
comfortable with an airline pilot navigating by dead reckoning. Si milar ly, managers who 
used to manage "by the seat of their pants" are now using key indicators to guide them and 
their processes. 

Quality principles are also beginning to affect government agencies. The popular 
book Reinventine Government: How the Entrepreneurial Spirit is Transforming the Piihlir 
Sector by David Osborne and Ted Gaebler 1 contains cases of government officials who have 
employed these principles. One of those officials is the newly appointed administrator of the 
National Aeronautics and Space Administration (NASA), Daniel Goldin. Mr. Goldin, a 
former program manager for TRW Space Systems, brings his success with industrial quality 
programs to the task of leading the nation’s space agency. In describing his vision for the 
NASA procurement system, Mr. Goldin said, 

We must have a procurement system which focuses on the customer -- a system in 

which each individual is empowered to do his or her job with excellence, with clear 

lines of authority and responsibility. 2 

The same can be said of any NASA function, including information systems (IS). 

Even before Mr. Goldin’s appointment, the NASA Johnson Space Center (JSC) had 
embarked on a quality program throughout its operation. Charged with planning and 
conducting space shuttle missions, JSC leaders set ambitious goals for reducing the time and 
cost of the missions. 

The Information Systems Directorate (ISD) is participating in the JSC initiative. ISD 
maintains the center’s computer facilities and telephone networks and assists end-users by 
acquiring and supporting workstations and software. ISD consists of 125 government 
employees who supervise thousands of subcontractor personnel. Together ISD and the 
contractor support data processing facility consist of IBM and IBM-compatible mainframes 
connected by an SNA backbone network. The network serves the majority of the 3,000 
government and many of the 15,000 contractor employees at JSC. ISD is also responsible 
for acquiring and supporting PCs, workstations and load area networks as well as the JSC 
telephone network and its computer connections to wide-area networks. 

ISD is establishing a new five-year support contract on January 1, 1993. The timing 
of this contract and the new NASA quality initiatives makes the investigation of quality and 
performance measurement timely. 


‘Reading MA: Addison- Wesley, 1992 

2 Domheim, Michael A. (1992) "NASA Hopes to Slash Contract Cost 5 to 10 % By 
Using TQM in Procurement," Aviation Week & Space Technology . August, 3, pp 22-23. 
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In order to get a more accurate picture of how performance measurement was being 
practiced, ISD asked a research team at the University of Houston-Clear Lake to study how 
industrial IS departments measured their performance. The research team conducted the 
study under a cooperative agreement between JSC and the university where government, 
academic, and industrial organizations cooperate on strategic issues in computing and 
information systems. 

The university study team had previously conducted a literature review on 
performance measurement for ISD. The review concentrated on the most important element 
of performance within the quality perspective - the customer’s perspective. 3 The study 
found a wide variety of techniques for measuring customer satisfaction documented in the 
literature. The most important techniques were customer surveys, executive interviews, 
service level agreements, and post-work complaints and compliments. IS organizations were 
also careful to align their customer focus with corporate goals, identify their own critical 
success factors (CSFs), and benchmark their performance against similar organizations. 
About 65% of Computerworld’s Premier 100 IS organizations did some kind of formal 
evaluation of customer satisfaction. 4 

Having concluded that study, ISD was interested in a broader study in two respects:' - 
expanding the focus to include all IS performance measures rather than just customer 
satisfaction and gathering information directly from industrial organizations rather than that 
found in the literature. 


OBJECTIVES 

The objectives of the new study were: 

• to understand the state of the practice in IS performance techniques for IS 
performance measurement 

• to gather approaches and measures of actual performance measures used in industry 

• to report patterns, trends, and lessons learned about performance measurement to 
NASA/JSC 


3 Bishop, Peter C., and Yoes, Cissy . Exploratory Study on Performance Measures as 
Indicators of IS Effectiveness. Houston TX: Research Institute for Computing and 
Information Systems (RICIS), UH-Clear Lake, May 1992. 

4 Maglitta, Joseph and Sullivan-Trainor, Michael L. (1991) "Do The Right Thing(s)," 
Computerworld Premier 100 . September, 30, pp 6-15. 



SCOPE 

Although the study is designed to gather information on performance measurements in 
other organizations, it is not technically a benchmarking study which would require a more 
intensive examination of detailed areas. Rather this study simply gathers information in 
preparation for the new support contract and for the performance measurement initiative. 


APPROACH 


Because of the complexity and depth of the information required, the study team 
elected to use face-to-face, open-ended interviews as its principal data gathering technique. 

An interview protocol was developed, a sample of corporations was drawn and the interviews 
were conducted. 


First, the study team developed an interview protocol that covered the major elements 
of IS measurement. The interview was designed to gather information on four aspects of the 
respondent’s performance measurement system: 

• a description of the current system 

• a history of how the system came to be 

• the respondent’s satisfaction with the system 

• expected improvements and developments in the system 

Interviewers also asked for recommendations that respondents might have for NASA as it 
embarked on its own performance measurement system. The interview protocol is contained 
in Appendix A. Respondents received an outline of the topics one week before the 
interview. 

Two interviewers conducted each interview. 5 One interviewer asked the questions 
(the questioner) and the other recorded the data (the recorder). The recorder asked clarifying 
questions at the end of the interview. The recorder transcribed the interview notes within 24 
hours. The questioner reviewed and edited the notes. 

The interview team also gathered documents on the respondent’s performance 
measurement system when they were available. Documents were given to the study team 
with the understanding that confidentiality would be maintained. 


5 A NASA representative also attended one of the interviews. 
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SAMPLE 


The most important criterion for selecting firms was their size. Large firms with 
substantial IS resources would be most comparable to the ISD environment. All firms in 
sample had more than one billion dollars in annual revenue. Characteristics of the IS 
division in each firm are contained in Table I. 

The firms selected came from a pool of 25 members of the Information Systems 
Research Center (ISRC) in the School of Business Administration at the University of 
Houston. The ISRC is a facilitating organization designed to bring together MIS 
academicians and practitioners on topics of mutual interest. One of the objectives of the 
ISRC is to study the development, implementation, management and utilization of 
information systems in organizations. 

The firms selected came from the following industries. 

- finance 

- oil and gas 

- utilities 

- waste disposal 



Table 1 


PERFORMANCE MEASURES - INDUSTRY PERSPECTIVES 

, DESCRIPTION OF THE ORGANIZATIONS 

COMPANY EMPLOYEES/ IS REVENUE SERVICE 

IS CUSTOMERS ORGANIZATION 

ABC • 7500 • 6 Executives • Direct chargeback to outsourcing 

contractor 

• 4000 employees • 7 company staff 
on LANS 

• 600 outsource staff 

EDF • 1356 • 200 staff • Control operations budget 

• 296 Branch • Control development budget 

offices 

GHI • 1356 employees • 199 staff • Control capital budget 

• Control development budget 

JKL • 40,000 employees • 850 staff • Chargeback covers operational budget 

• Additional R&D budget 

• Customers free to purchase outside 

services 

MNO • 26,000 employees • 280 staff • Development budget 

• 400 locations • Operations budget 

PQR • 26.000 employees • 170 staff • Allocation algorithms 

• Full-cost recovery for operational budget 

• Control capital budget 


All firms used large data processing facilities. One respondent served in an 
applications development and end-user support organization only. They purchased their data 
processing from a subsidiary of their parent firm. As a result, that respondent was unable to 
give us information about central data processing measures. 

All of the divisions had some responsibility for end-user computing, though two firms 
allowed their customers to purchase goods and services from outside vendors. Three also 
had responsibility for the telephone operations in their firms. Finally, all IS division heads 
were either corporate officers (vice-presidents) or they reported to them. 
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Organizations varied on the degree to which their customers 4 were autonomous from 
the central IS function. Two firms maintained highly centralized operations in which all of 
the IS resources were managed within the IS organization. The IS organization managed the 
capital budget which it allocated after consultation with its customers. While some 
outsourcing does take place, it is usually done in conjunction with IS planning and assistance. 

In contrast, one firm’s business units maintained their own application development 
units and could purchase IS products and services directly from outside vendors. The IS 
organization, therefore, had to compete for customer dollars with all the vendors in the IS 
industry. 

Five organizations delivered the bulk of their products and services themselves. One 
firm supervised a large outsourcing contract. The vendor owned and maintained the firm’s 
data processing facility for all but the most critical IS operations. The outsourcing 
relationship dramatically shaped that firm’s approach to performance measurement. Since 
they could not control performance or its measurement on their own, the company initiated 
extensive and fruitful negotiations with their incumbent contractor. The process was 
complicated because the original contract made no explicit provisions for quality 
improvement. 

As a result, the development process for their performance measurement system was 
extensive because the details were crucial both for the company and its contractor. Both 
negotiated in good faith and came to an agreement that supported the quality of the IS 
function while being fair to both parties. The negotiations began with contractor measures 
applied to company data which provided the baseline on the historical level of service. The 
parties then developed their definition of minimally acceptable and outstanding service. They 
used a third-party negotiator to help reach agreement. 

The result is the most explicit and detailed performance measurement system in the 
sample. This case most closely parallels the NASA context. Although NASA owns its 
facilities wiiich this firm does not, both work closely with a large contractor force to provide 
service to IS customers throughout the organization. One critical element which the 
respondent stressed was the need for mutual trust and respect across the contract. 

While negotiating new performance measures with a contract already in place is not 
ideal, this firm has shown that it can be done and that it can enhance the working relations 
between customer and contractor. What was an arms-length business deal is moving to a 
genuine partnership. 

Three organizations charged their customers for services. In the firm with the major 
outsourcing contract, charges went directly from the contractor to the business unit. The 


6 In this report, "customers" refers to the IS customers which are users and organizations 
in the same corporation. No IS organization served the corporation’s final customers 
directly. 
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other two chargeback organizations funded their budgets through the delivery of products and 
services. One firm does not charge customers directly, but they do report usage and costs to 
each application owner and each line manager once a year. The costs were based on the 
total IS budget times the proportion of resources used by each application and by each 
organization. Though not charged directly, managers were able to tell how much of the 
corporate resource they were using. 


GENERAL FINDINGS 

Requirements for a Successful Performance Measurement System 

The most consistent finding across all the interviews is how difficult it is to institute a 
successful measurement system. All respondents reported that the firm needed a different 
mind-set to use performance measurement for management and quality improvement over 
more traditional practices. Every respondent indicated that their organization had extreme 
difficulty changing the mind-set of the organization. Respondents frequently returned to the 
themes of "cultural change" and "paradigms" whenever the topic turned to implementation. 

t 

The problem of implementation was not technical. Given some baseline da ta and a 
team of willing members, performance measurement systems can be developed. Only one 
respondent attributed some difficulty to the technical details of the system. Even at that, 
changing the mind-set of the firm and the contractor was still the key element of success. 

For a successful implementation, the following changes were recommended by several 
respondents: 

• Management by fact is a term that describes a management style that relies on 
consistent and valid measures to make decisions and guide actions. Two 
phrases that describe this management philosophy occurred in more than one 
interview: 

If you don’t measure it, 
you can’t manage it. 

Things which are measured improve; 
things which are not measured don’t. 

• Measurement should begin and end with a customer focus. Technical 
measures of machine performance are important for delivering service, but 
they are only the means to the end. The measurement program should 
concentrate on whether IS helps customers meet their goals. 
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• Organizations and teams must have goals to which the measures are linked 
before meaningful measures can be developed. Measures that are simply "nice 
to know" were usually ineffective and soon discarded. 

• Measurements should be developed by and for the people being measured. 

They know the process better than managers and consultants do, and they are 
more affected by the results. Employees have to see that the measurement 
program benefits them directly in their work for them to spend the time 
keeping it up. 

• Measurement takes time and money. Large firms were spending up to .50 
FTE (full time equivalent) compiling the hardcopy reports on their data 
processing centers alone. Customer surveys need to be developed, interviews 
need to be conducted, function points need to be counted, data needs to be 
analyzed and interpreted. Measurement is not cheap. 

• Measurement must be followed by visible decisions and actions based on the 
measurement. If the measures don’t make any difference in anyone’s 
behavior, they soon lose their importance and support. Something important 
must be riding on the outcome of the measure. 

Overall the cultural change accompanying performance measurement was singled out 
as the most important prerequisite for a successful system. Measurement systems either 
accompany or follow organizational change. They are never simply an additional item to an 
existing operation. 


Trends in Performance Measurement System 

Past experience and current plans of major IS organizations indicate that a pattern of 
change and new measurement strategies are emerging. 

Automatic measurement 

One of the clearest trends for the future is the move to more automatic measuring 
systems. Machine instrumentation is no doubt more reliable and less costly than manual 
methods. Thus SMF and MICS data on mainframes are the most sophisticated and mature IS 
measurements. Automatic processes are expected in software development as the result of 
adopting CASE architectures. Network management tools are developing rapidly and may 
soon be able to identify problems at the customer’s workstation that are effecting availability 
and performance. 

Experience showed that manual measurement systems were harder to keep up. 

People do not consistently log data on their service requests or development tasks, for 
example. Often they simply do not have the time to document their work. 
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No respondent, however, recommended that automatic measurement was essential. 
Some measures, even manual ones, are better than none. Manual measurement was 
successful as long as it was attached to a business goal and people understood why it was 
important and what they would gain from the process. 

Individual an d team ownership 

One firm responded to the problem of collecting measurement data with an extensive 
training program on performance measurement~not only with how to collect the measures, 
but why collect them. When the employees realized how helpful it was to know the volume 
of requests, the average time to resolution, and the number of open requests, they kept the 
measurement system up-to-date themselves. 

Another firm is using an aggressive teaming strategy to meet the challenge of IS 
performance. As a result, the performance measurement system is migrating to the team 
level as well. Work teams develop, collect and use their own measures themselves. They 
do not necessarily report them to the central office. The IS office will continue to monitor 
the division’s performance through customer surveys, but individual activity measures, 
except for data center operations, are not as visible to the central office as they used to be. 

The lesson here is that measurement systems compiled for someone else’s use (i.e., 
management’s) usually fail. If the measure takes time, then the individual expending that 
time must see some direct benefit in their work. 

Fewer and more focused measures 

Another trend is to reduce the number of measures which firms rely on. Automatic 
measurement often generates too many measures. Firms are trying to link their measures to 
business goals and to establish reasonable criteria for success. A few well-thought measures 
are better than a haphazard array. 

Summarizing multiple measures 

Summarizing even a few measures for quick assessment or management monitoring is 
always a problem. Some companies used averages or other more complex techniques for 
combining measures. A more common measure, however, was the percent of times the 
measure met the criterion. — 

For higher level summaries of the data, firms are moving to report cards (one overall 
assessment of the total area) and exception reporting (times that the measure failed to meet 
the criterion). The on-line system for data processing measurement in one firm combined 
both of these trends. Any score below its report card value of 100 indicates that a failure 
occurred. The user can then drill down into the data to find out where the exception 
occurred. 
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Detailed Problems and Lessons Learned 


Respondents also identified perennial problems and lessons learned with performance 
measurement (Table 2). The most serious problem is collecting manual data. It is difficult 
to establish procedures which are accurate and cost effective. Some firms even abandoned 
manual systems because of these problems. Those which succeeded were developed by and 
for the people who were to collect the data and were supported by sufficient training to 
insure consistent operation. A second problem mentioned by more than one respondent was 
the issue of getting the right number of measures and the right level of detail. Too many 
measures can be as meaningless as not having enough measures. By the same token, striving 
for maximum measurement precision is an unnecessary burden on the measuring process. 

The final difficulty was to keep the measures current against the changing needs of 
the business. Two firms had just been through a major corporate restructuring and one firm 
was experiencing one as we spoke. These events create new visions and goals for the 
organization, the progress toward which needs to be measured. The lesson is that no 
measurement system is static. Solving the perennial problems and serving a dynamic 
business require measures to be flexible throughout their lifetime. 

Firms also shared a few specific lessons which they learned in designing and 
implementing their measurement systems. (Table 2) The most important lesson is that the 
human dimension of the measurement system is more important than the technical details. 
Finding out the needs of the business, getting and keeping everyone involved, reporting 
results back to the customer — all of these require attention to the human relations aspect of 
performance measurement. Firms also reported some lessons on the technical side: daily DP 
data is more useful than monthly summaries; begin the design process with some benchmark 
data; use consultants to facilitate joint design sessions between companies and outsourcing 
contractors. The firms also admitted that the process of establishing a performance 
measurement system takes time, and attention to details is one of the criteria for success. 



Table 2 



PERFORMANCE MEASXJRES-1ND USTRY PERSPECTIVES J 


DETAILED problems and lessons learned 

COMPANY 

PROBLEMS IDENTIFIED 

LESSONS LEARNED 

ABC 

• Perception of Details 

• Use consultants to develop comparison studies 

• People don’t want to spend money to tell good « 
news 

• Maintain in-house systems which are critical to 3 
mission 

• Tangible evidence is important 3 

• Use creative adaptation of what is available 

DEF 

• People who don’t know what the data is for are 

• Use consultants to kick off customer and 5 


not motivated to collect and use it 

employees surveys 

GHI 

* Difficult to get data unless it is embedded in the 

• Monthly data not as revealing as daily data — 


process Sc comes out automatically 

• Lack of definitions about what is right outcome "| 


• Motivation 

► Ambivalent about productivity 1 

►Traffic St Quality = 


• Content 

if 


• Budget 

II 


• Lack of the "Big Picture* 

I 

JKL 

• Metrics hang-on to past usefulness 

• Easy to develop Sc use metrics in stable 

• Include vendor as part of the process for setting ~ 
goals and metrics ^ 


technology but almost impossible with new 

• Under force of new business organization, m 


technology 

metrics have been pushed down the organization = 
just like everything else || 

MNO 

• Progress can be faked with a flurry of metrics 

II 

• Metrics don’t need to be more precise than the “ 
decision or actions that depend on them. Ultra- m 


• Need buy in from the person being measured 

precision may be too costly 

• People who know the process can usually ™ 

design the metrics better than anyone 

PQR 

• Measures proliferate without meaning 

1 

• Satisfy first the executive who pays the IS bill {§ 

• Feedback data on performance to customers as | 
a way to strengthen partnerships 















PATTERNS BY PRODUCT/SERVICE CATEGORY 


The interview team also discerned some patterns by the IS area being measured. 
Performance measures occurred in four categories. The areas covered here are customer 
satisfaction, data processing, application development, and end-user computing. 


Customer Satisfaction 

Customer satisfaction is the most important attribute that an organization has in the 
current TQM environment. The longer organizations evolve their performance measures, the 
more they realize that the customer is the final arbiter of all quality issues. That insight does 
not come right away, but rather only after a long period of dialogue with the customer and 
trial-and-error with IS improvement. Only then do customer partners become an integral 
part of the IS business. 

Most firms begin their customer satisfaction measures with a customer survey. 

All respondents reported that designing and conducting customer surveys was more difficult 
than it appeared. They also wanted their survey to be fair and anonymous, particularly the 
first time they delivered it. As a result, all initially contracted with a consulting firm to help 
them develop their survey material. After that experience, most were able to continue the 
process on their own. 

Organizations typically passed through a series of stages for their customer measures 
which is described more fully in Appendix B. The first round of customer surveys usually 
surfaced more problems rather than providing constructive help. Customers "dumped" all 
their frustrations with computing on the IS organization. After that stage, however, IS 
organizations began to initiate real dialogue for improving their operation and assisting the 
customer in meeting their goals. 

Part of the dialogue with the customer was the time taken to report the results of the 
surveys back -- either in hardcopy publications, on-line systems, or management briefings. 
Verifying what IS hears from the customer is an excellent way of confirming that IS is 
listening and correcting any misunderstandings that may arise. 

Furthermore, customer input also had to create a visible change in IS performance. 
Customers who see real change will be more ready to offer useful feedback in the future and 
work with the IS organization for mutual benefit. The final stage is a partnership between IS 
and its customers for the benefit of both. Customers can do their job better with a well 
performing IS organization, and IS has loyal customers on their side when assistance is 
needed. Continuing dialogue toward even more improvement commences at that point, 
keeping the relationship alive. 
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Customer satisfaction-examples 

One firm is considering combining a manager and an end-user survey into one 
instrument. The combined instrument may be the basis for providing financial incentives for 
contractor performance. Financial awards will be awarded by the customer unit when it has 
surplus funds, and when the contractor has earned an excellent rating on the satisfaction 
measures. 

A second firm uses both a customer and an employee survey to gather data on its IS 
performance. They report receiving between 35% and 40% return on the customer survey, 
indicating a lot of interest in responding. The employee survey also generates wide 
participation. It measures employee satisfaction on 1 1 attributes, asking for a performance 
and an importance rating for each attribute. 

Two firms conduct periodic interviews with its clients to determine their satisfaction 
with its performance. The interviews at one firm concentrate on the behavior of the IS 
personnel more than the systems they provide. The attributes they measure are: 

accessibility 

responsiveness 

problem solving ability 

technical support 

on-time delivery 

reliability 

attitude 

competence 

communication 

understanding requirements 

administrative practice 

value 

Customers are asked how important each attribute is, how well the organization has 
conformed to requirements and how much it has improved or declined. They are also asked 
for items that are done particularly well and suggestions for improvement. Finally, the 
customer awards an overall performance rating from 1-5 with an explanation for the rating. 

Another approach is found in one IS organization whose goal is to be a business 
partner to their customers. They define their customer both as the business unit executive 
with financial responsibility for computing and as the individual end-user. Each unit also 
employs a business information manager, a member of the customer organization, who is 
responsible for maintaining the quality of IS service to that unit. 

The business information manager assesses the quality of that service annually in a 
series of meetings with the executive and computing teams in his unit. The structure of the 
assessment is open, but IS suggests topics such as areas that are well done and those that 
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need improvement, key questions and expectations for the next year, and a management 
message -- i.e. , the overall impression which the business unit wants to leave with IS 
management. 

The business information manager finally awards a three-scale score (are delighted 7 , 
meets requirements, doesn’t meet requirements) on four attributes (responsiveness, 
effectiveness, professionalism, and understanding the unit’s business). They arrive at that 
score after review with the senior business executive. The scores and other pertinent data 
are reported to IS management during a "read-out" process that covers several weeks. The 
read-out opens the strategic planning process for the coming year. Customer executives and 
computing teams also receive a briefing on the results of the assessment and planned focus 
for the new year by the business information manager. This approach is classical industrial 
marketing -- concentrating on the customer who pays the bills. 

Their measurement system continues to evolve, in fact going against the trend to 
reduce the number of measures. The current system focuses on three service areas 
(planning, new capabilities, existing capabilities). They will expand that next year to eight 
areas for more detail. They are also likely to expand the number of attributes measured. 
Finally, they are instituting a series of periodic quarterly checks to capture more frequent 
data. The system had been pared down too much, and they are readjusting to a more 
moderate level. Overall, however, the system provides IS with high level customer feedback 
to select goals and focus areas for the coming year. 

In sharp contrast to the increasing emphasis on customer satisfaction, one firm 
recently discontinued its customer survey to save money. In the midst of downsizing, they 
did not feel that they were getting sufficient value from the survey to warrant its cost. 

Another firm has never formally measured its customers’ satisfaction. The department 
already knows where the problem areas are and believes that a customer survey might 
increase expectations beyond what they can deliver, thus making attitudes worse. Customer 
satisfaction is recapped in Table 3. 


7 A "delighted" rating means the organization delivers on requirements that the customer 
did not ask for or expect (e.g., not only getting a good deal on a rent car, but having it 
delivered to your house, too). 
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Table 3 


PERFORMANCE MEASURES - INDUSTRY PERSPECTIVES | 

FIXINGS-CUSTOMER SATISFACTION | 

COMPANY 

EXAMPLES 1 

ABC 

• Manager and end-user survey combined 

• Report card being developed 

• Possible financial incentives | 

DEF 

• Customer and employee surveys 

• Consultant initiated 

GHI 

• None 

JKL 

• Customer Surveys 

• People focus more than machines 

MNO 

• Survey discontinued 

PQR 

• Executive focus 

• Business manager in customer organization 


Data Processing 

The mainframe oriented measures are almost uniform throughout the industry. They 
have been around for decades, automatic tools exist which generate the data. Most of the 
measures come from SMF type data logged on IBM mainframes. 

Benchmark databases and services are available for those wishing to compare 
themselves with other organizations. Two of the firms benchmarked their operations against 
industry peers annually using the Real Decisions 8 database. One firm has used Real 
Decisions occasionally, and one collected the benchmark data themselves. 

The most important data processing attributes measured are availability and response 
time to customers. Firms developed criteria or targets for those few measures which were 
considered crucial. Firms also record trend data on many other attributes. The data is 
usually packaged into hardcopy reports and distributed to IS and customer management on a 
monthly basis. One firm is reorganizing its report which was organized by department to 
one which organizes the data by product/service category to increase its customer orientation. 

Two respondents reported that the daily DP data was more useful than the monthly 
statistics. The monthly data either hid important short-term variations or they were always 
so high that they became meaningless. 


* Real Decisions Corporation, Darien Connecticut 
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Data Processing-examples 


The company with the outsourcing contract had the most detailed set of performance 
measures as a result of the long negotiation with their outsourcing contractor. They divided 
their services into 17 service levels, each measured with a service level agreement. The 
agreement includes a format for each of the performance measures of that service (Table 4). 
The format consists of general descriptors for each service and one or more levels for that 
service. For example availability of a system would be the service, and 98.5% of the time 
the level. 

The agreement contains special provisions for the quality of the central data 
processing services. They identified nine categories of service level agreements here. The 
categories cover availability, response time, turnaround time, and various management 
functions. Exact levels are specified for each category for each of the computing 
environments they support. 

The agreement does not cover existing application systems. Application managers are 
reluctant to spend the time and money instituting and supporting measures at the application 
level. They have fairly good systems running today and do not feel the need to verify that 
further. This belief that measurement is unnecessary for smoothly running systems was 
common in other firms as well. The problem is how to keep an emphasis on improvement in 
systems that have apparently maximized their performance. 

No financial penalties are imposed when systems fail to meet their minimum criteria. 
Establishing blame is not only difficult, but contrary to the partnering mentality that the 
company wanted to promote. Firms look to find why the process failed and move to fix it as 
best they can. The result is an environment where everyone is striving to do better each 
time. 


One firm has been putting its DP data on-line for 10 years. Division policy 
emphasizes openness of results throughout the corporation. The on-line reporting system 
tracks the daily performance of two data centers. Statistics are reported for each center and 
for each environment (TSO, CICS, etc.). The report contains a summary score for that 
center and environment. The score is based on availability and response criteria for that 
environment. If all criteria are met, the score is 100. Points are subtracted if any measure 
falls below the criterion for any hour of the day. Hourly histograms are provided to show 
where the failure occurred. Owners of processes that fail below the criterion must file an 
incident report on the reason for the failure. 

They are now moving into a second-phase of measurement where their centralized on- 
line systems are being retired in favor of decentralized measurement by work teams. The 
on-line systems driven by machine measurement are still active, but most of the central 
measurements that required manual processing have been abandoned. 
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Table 4 

Performance Measure Format 


* 

General descriptors for each service — 

* 

Service description: 

Definition 

m 


Activities in that service 


Service environment: 

Geographic location 

i 


Technical environment 


References: 

Related documents 

i 

Service availability: 

Open for business 


Report frequency: 

Daily, weekly, monthly ... 


Specific descriptors for each service level -- 

m 

Service level criteria: 

A statement of the desired level of performance for 

m 


that service 


Service level: 

The percent of times the desired level is to be met. 

m 


For non-machine criteria, the service level is 



expressed as a minimum level (required) and a 

w 

- 

target level (desired) 

V 

Severity code: 

An indication of the importance of that 

m 


criterion: 



1 - significant impact 

S 

9 


large number affected 



2 - minimal impact 

== 


large number affected 

B 


3 - minimal impact 



moderate number affected 

s 

Measurement method: 

The measurement technique or source of 

■ 


information for the criterion 

m 

9 


. ... • - 
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The monthly hardcopy report for one organization concentrates on resource utilization 
organized by application and by customer organization. The IS budget for CPU and DASD 
is divided by the application and organization use rendering a cost figure for each application 
and for each organization down to the individual end-user. Customers are not responsible 
for funding those costs (i.e., no money changes hands), but they do become aware how much 
they use the corporate resource. The cost data is also used to benchmark cost ratios against 
similar organizations in the industry. 

Another hardcopy report highlights three attributes for special attention: safety 
(number and hours since last lost time), availability/reliability (percent up-time by 
environment) and customer service requests (number and call waiting time). The report also 
contains trend graphs on other system parameters. The organization is beginning to use 
statistical process control (SPC) limits to begin to measure the performance in these other 
domains. 

This same data processing group has also negotiated a series of five quality measures 
with one of their computer service vendors. The report on this system generates a scorecard 
every month that results in a yes/no score on whether the system met the requirements. 

Finally, the corporation benchmarks its costs against the Real Decisions database each 
year. Their goal is a 10% improvement against the industry average each year (in the face 
of an 18% improvement in the average itself). 
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PERFORMANCE MEASURES - INDUSTRY PERSPE 






















Application Development 

The application development departments (Table 6) are experiencing the most change 
in performance measurement of any IS function. Not only is software development in 
general experiencing rapid change, but the concepts and tools for getting beyond source lines 
of code (SLOC) are appearing. 

Two firms have measured the quality of their in-house systems by tracking the 
performance of those systems in the field -- specifically measuring the number of ABENDS 
the system generates in its first quarter or year. During development, the most common 
measure is the number of incident or discrepancy reports per unit. 

Many firms are planning to measure the function points in their systems. Analyzing 
the existing systems is a mammoth task, but they expect to gain more knowledge about their 
application development operation as a result. Respondents often mentioned the International 
Function Point Working Group (IFPWG) as the center of this movement. 

Once function points for each system are available, or ganizati ons are then able to 
calculate the standard quality, schedule, and cost variables per function point. Some firms 
have adopted this strategy; others are collecting baseline data in preparation for adopting it; 
others are still studying the issue. But the trend right now is toward function point units as 
the basis of application productivity measures. 

Application development - examples 

One company has a separate system for quality management of its applications 
development. They use periodic quality assurance reviews. Every major development effort 
is reviewed each quarter and at each major life-cycle milestone. The review is led by an 
independent reviewer who studies progress on the project and delivers a report to the 
executive sponsor, an IS representative, and the project manager. The manager also prepares 
a risk assessment and mitigation memorandum for the review. The review concludes with a 
written report on the status and risks for the project over the next period. The process does 
not produce any reportable metric except that the review took place. 

Another firm captures personnel effort by project and by activity in order to measure 
staff effort in development, support, and administration in application development. 

A third firm measures a few fundamental parameters of their work: the availability 
of the application, the proportion of times the application runs without error or interruption, 
the variance between planned and actual completion time of projects, and the timeliness of 
handling service requests. Each of these parameters is measured on a monthly basis and 
tracked using SPC control limits. Any values which fall outside of the limits are checked for 
improvement. (The timeliness parameter has not been implemented because they are having 
difficulty capturing when service requests begin and end.) 

One applications development group is engaged in a three-pronged effort to increase 
its quality and performance. They had used an individual reporting system against a project 
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work-breakdown-structure (WBS), but the system was too burdensome to maintain. The first 
part of the new effort uses the Malcolm Baldrige criteria to grade each team’s performance. 
The second aspect is to gauge the size and cost of their existing systems using function point 
analysis. Finally, they are participating as a research partner with a university and a vendor 
to try a set of university-developed tools to gather baseline productivity data on new 
applications. They intend to establish some type of permanent performance measurement for 
application development and support once they accumulate two years of baseline data. The 
emphasis is on enhancing development productivity. 


This firm has developed an on-line system to grade specific applications. An analyst 
would rate the system on stability, responsiveness, flexibility, usefulness, and general 
satisfaction. Most applications have not been graded for a number of years, however, and 
the system is largely abandoned. 

Many IS organizations execute a detailed service level agreement with each customer. 
The agreement covers the support of existing systems as well as the development of new 
ones. It not only spells out the work to be performed, but the measures used to judge the 
quality of that work. The firm then rates the quality of its application development on how 
well it performs on the service agreement. 

One company reported that service level agreements used to be hard to meet, but as 
the organization has improved, all measures are met routinely. Productivity measurement of 
application development is based on a major study which calculated the function points in 
175 existing systems. The organization undertook the study to justify the cost of moving to a 
CASE environment. The organization made the move, and are now positioning themselves 
to have the CASE tools automatically generate the function points in each new application. 
Once function points have been determined, the organization then calculates cost, schedule 
and quality measures per function point. The default quality measure is the number of 
incident reports on a finished system per function point. 

The company believes that the uniform CASE environment will someday automate the 
process of productivity measurement. The CASE tools illustrate that the benefit of 
automating a process may lie as much in the information which the tools provide as the time 
that it saves. 

Another application development group is reconstructing their performance 
measurements for application development following their reor ganizati on. They are 
developing a new system with a radically different orientation. The purpose of the old 
system was vertical management control. Now they are using a horizontal system that 
emphasizes real-time process data. The old system tried to measure everything where the 
new system is oriented only to critical business issues (e.g., reduce cycle time, cost, defects, 
delays, etc.). Those collecting the measures did not have any influence on the measure in 
the old system. Now everyone involved must sign-off before the measure is implemented. 
Each measure also requires extensive documentation before approval. 
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including: 


critical business issue 

measures/goals (specific change to a product or process) 

critical processes (which processes are affected) 

process owner 

action plan 

estimated costs 

agreement by all parties 

The action plan for the measure includes items on data collection, analysis, and reports, 
including typical tables and graphs. 

The new system is under development at the current time. It is an attempt to combine 
performance measurement at the team level while retaining visibility at the corporate level. 

Two firms mentioned that they do not have productivity measures based on work 
units right now, but that they expect to in the future once they are comfortable with their 
understanding of what would be an appropriate measure. 
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Table 6 


PERFORMANCE MEASURES - INDUSTRY PERSPECTIVES 

APPLICATION 

COMPANY 

EXAMPLES 

ABC 

• Quality reviews for each major project each quarter 

EDF 

• Time & effort reports by project 

• Focusing more on quality and defects 

• No productivity measures 

• Used Real Decisions occasionally 

GHI 

• A few statistics gathered after deployment (ABENDS, etc.) 

• Monthly trend against SPC controls 

JKL 

• Team measures 

• Function point analysis 

• Productivity analysis 

MNO 

• Detailed service level agreements with customers 

• Function point analysis 

PQR 

• Complete plan and approval for each measure 

• Must address a significant business issue | 















End-user Computing 

No one has a handle on this rapidly changing technical area. First workstations, now 
LANs are exploding throughout the corporations. Many respondents did not think the 
systems could be measured until the technology stabilized and tools were developed for 
tracking activity and performance. 

The best that most companies could do was the timeliness and quality of their service 
requests. One company tracked time to respond to problems, average age of problems, and 
number of calls put on hold. Two companies also tracked the time that a customer had to 
wait on the telephone as a measure of performance. 

One firm had initiated a study to fully measure the cost of installing and operating 
local area networks (LANs). They suspect that mainframe revenue has partially subsidized 
those costs. 

Measuring end-user computing performance is even more difficult in a strongly 
customer focused organization. Some companies reported outmoded measures such as 
availability of a particular region or application on the mainframe as a measure of end-user 
availability. That measure obviously neglects the performance of the network. Network 
performance measures are now available, but they still cannot tell whether an individual PC 
is turned on and functioning properly. One IS manager estimated that he would have those 
tools in two to three years, but until then, he was not satisfied that he had a good approach 
to validly measuring end-user performance. 

CONCLUSION 

Overall, respondents were quite satisfied with their performance measurement systems. 
Table 7 lists the specific satisfaction by organization. TTie one exception was a firm that had 
not been collecting performance measures very long and had not devoted significant 
resources to the task. They viewed their system as "OK," but it did not figure strongly into 
the organization’s goals or improvement strategies. The overall satisfaction, however, did 
not deter firms from continuing to improve their already good processes. As mentioned 
above, they were trying to reduce the number of measures they collected and link them more 
closely to the business goals. Some organizations had given work teams the responsibility 
for collecting most of the measures, and others were continuing to strengthen their ties with 
their customers. 
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Table 7 


PERFORMANCE MEASURES - INDUSTRY PERSPECTIVES 

SUMMARY & CONCLUSIONS 

COMPANY 

SATISFACTION 

ABC 

• Good start with utility measures 

• Quality assurance has worked well 

• Customer satisfaction believed on right track 

DEF 

• Fairly satisfied, particularly with surveys 

• Concentrating more on quality measures 

• Trying to reduce number of measures 

GHI 

• Low degree of satisfaction 

• Little impact on behavior or process improvement | 

JKL 

• Quality initiatives successful II 

• Reversal of negative IS image I 

• Moving measures to team level | 

MNO 

• Good progress 

• More work to be done especially in stockholder satisfaction 


• Rebuilding measures after reorganization 

• Quite satisfied so far 

• Continuing to improve 


RECOMMENDATIONS TO NASA 
From Industry Respo ndents 

The interview team asked each respondent about their spe cific recommenda tions to 
NASA as they embark on the process of redesigning their performance measurement 
systems. Each organization which had specific recommendations shared a slightly different 
orientation. 

One respondent spoke of the need to establish a good working relationship with their 
major contractor. They found that they were more successful in measuring performance 
when they formed an equitable partnership based on working together to achieve a high 
quality IS organization: 

• establish a relationship based on the desire for quality 

• strive to be fair to both parties at all times 

• start with some historical data as the basis for negotiation 

• define terms carefully 

• expect a long process 
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A second respondent reflected on the need to plan and develop the performance 
measurement system carefully and to review it often: 

• plan the measurement system; don’t just let it happen 

• people who carry out the process can design better measures than technical 
consultants or managers can 

• keep the focus on improvement rather than catching someone doing something 
wrong 

• too many measurements are as unhelpful as too few 

• use measurement for improvement and drop the measure once the 
improvement has taken place 

• periodically review the system and eliminate obsolete measures 

• measures do not need to be any more precise than the decisions and actions 
that depend on them 

• raw data and tools for analysis are better for executives than summarized 
reports are 

Another set of recommendations focused on the overall purpose of the performance 
measurement system and what NASA could expect to find when they are finished: 

• developing adequate measures requires long-term commitment 

• concentrate on measures which are important to the customer 

• measurement goal is productivity not policing 

• measurement usually results in re-engineering the process 

• use outside groups to benchmark performance 

A final list emphasized the customer’s perspective and role in deciding which products 
and services to develop and to measure: 

• ask customer group what their products and services are and what focus IS 
should have 

• ask the IS organization to develop a marketing brochure (TIP: if it looks like 
an organization chart, it is not customer-oriented) 


Developed From Research findings 

* strong, consistent executive level commitment is required for the 
development of a performance measurement system 

* form a performance measurement working group to oversee, facilitate and 
coordinate development efforts 

* budget adequate resources and time to develop and implement the system 
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* provide for meaningful and adequate training for employees so that they 
can understand the system and fully participate in its development and 
implementation 

* use a team-approach to developing measures, including the people most 
involved with the area to be measured 


* include vendors in performance measurement development 

* use a customer-oriented focus for the performance measures 

* use performance measures to support improvement 
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APPENDIX A: PERFORMANCE MEASURES INTERVIEW 


Organization: Representatives: 

Interviewers: Peter Bishop, Cissy Yoes (UH-Clear Lake) 

Location: 

Date, Time: 

Purpose of the Interview: 

The purpose of this interview is to obtain a description of how your organization uses 
performance measures, what measures you use, and why you consider them important. This 
information is being gathered to help the Information Systems Directorate at JSC/NASA in 
the redesign of their performance measures. During this interview, our objectives are the 
following: 

1) Introduce the study we are doing. 

2) Obtain a description of your performance measurement system and how you 

designed and implemented it. 

3) Understand what your performance measures are, how you use them, and how 

satisfied you are with these measures. 

4) Share a performance measures model that we have developed for this study and 

solicit your comments on the model’s validity and usefulness. 

5) Obtain any documents you are willing to share with us, such as, consulting 

reports, metrics, process descriptions, and internal reports at various levels. 

Agenda: 

Introduction 
-- UHCL project 

— confidentiality requirements 

Description of Information Systems Organization 

— mission, goals, scope 

— organizational structure 

— operations 

Overview of Performance Measures 

— design 

— implementation 

— operations 
-- reporting 

Assessment of Performance Measures System 

— satisfaction 

— problems 

-- lessons learned 

— recommendations to others in development stage 
UHCL Performance Measures Model (may be discussed earlier 

at respondent’s request) 

— overview 
-- comments 

Closure 
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APPENDIX B: THE LIFECYCLE OF PERFORMANCE MEASUREMENT 

Firms did not directly articulate a life-cycle for measurement systems, but piecing 
their stories together shows there appears to be one. Every firm has a distant past when they 
took no measurements and, in fact, hardly considered them. The situation was analogous to 
end-user computing today. The rapid rate of technological change and the lack of standard 
tools made measurement difficult if not impossible. That situation existed in mainframes 
before SMF, MICS, and Real Decisions came along. We will call it Phase 0. 

Phase 1 begins when firms first consider doing some measurement. They feel the 
need to improve the system. Something is wrong, and the measures are developed to find 
out what it is and how to fix it. Ironically, many firms stop at this phase. They expect 
negative results from their initial measurements. They are afraid that the results will put 
them in a bad light in the eyes of upper management or that their customers will vent their 
frustrations all over them. They say, "We already know what the problems are. We don’t 
need measures to tell us that. " 

In fact, initial measurements usually do show some serious problems. (That’s what 
they were instituted for!) And customers, long carrying their frustration in silence, do 
unload on the first person who asks them what they think of the IS department. But no one 
has reported serious injury from these encounters (except perhaps to their egos), and what is 
more important, the measurements and the feedback actually improve. That’s Phase 2 - the 
improvement phase. 

Phase 3 occurs at the critical juncture when the measures plateau at a high value. 
Measures such as 99.8% availability, 0.2 sec response time, and 97% customer satisfaction 
are hard to beat. Interest in measurement wanes at this point. Some firms keep increasing 
the standard to keep effort directed on the task. Some change the measurement strategy to 
some other attribute. Some recommend dropping the measure altogether and moving on to 
other topics. Others change their whole approach to measurement, adopting the team 
approach or the executive interview as the primary measures. 

The ironic conclusion to the lifecycle analysis is that firms who measure heavily 
believe that they have a moderate problems. Firms with problems that are too big are afraid 
of the results. Firms with problems that are too small cannot justify the time and cost of 
measurement. Both will have to measure sometime, however. The first group" must 
ultimately improve or go out of business, and the second group will ultimately adopt new 
goals that require measurement and improvement. The result is that measurement comes 
sooner or later to every organization. 
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APPENDIX C: A TAXONOMY OF PERFORMANCE MEASUREMENT SYSTEMS 

The first task is to explain the nature of each performance measurement system in 
terms of industry or corporate variables which affect the IS organization. The sample is too 
small to test these relationships statistically. Nevertheless, a small set of variables do 
describe each firm’s system in terms of its larger context. Furthermore, these variable might 
be the subject of continued research with a larger sample. 

Table 8 

Explanatory Variables for Performance Measurement Systems 


Company 

Cost 

pressures 

Quality 

program 

Restructured 

organization 

Corporate 

culture 

Result: 

PM System 

MNO 

Yes 

(internal) 

— 

— 

— 

Benchmark 

CASE 

ABC 

Yes 

(outsource) 

— 

— 

— 

Detailed 

agreement 

GHI 

No 

No 

— 

— 

Few 

measures 

EDF 

No 

Yes 

No 

— 

Defects, 

satisfac. 

JKL 

No 

Yes 

Yes 

Decentral 

Team 

ownership 

PQR 

No 

Yes 

Yes 

Central 

Executive 

focus 


The primary explanatory variables are listed in Table VII. This table does not explain 
everything unique or interesting about the various firms described, but it does capture some 
of the major items. 


The first variable describes whether the IS organization felt that they were under cost 
pressures. Two organizations did feel that they were, but they adopted different strategies to 
deal with that pressure. One firm reduced its headcount and monitored its industry closely. 
Its objective was to reduce costs and keep them down. One of their measurement strategies 
was to benchmark industry costs and use that to justify their annual budgets. They also 






























moved aggressively into a CASE development environment which, though initially costly, 
was justified on a cost-savings basis. 

The other response to cost pressure was to outsource the IS function. The resulting 
performance measurement system was the most detailed of the sample. Firm and contractor 
chose to develop this level of detail in order to reach their mutual quality objectives and still 
be assured that the system was fair to both sides. 

In this model, cost pressure preempts the other variables. While firms under cost 
pressure also have quality programs and other organizational experience, they do not seem to 
impact the type of performance measurement system they adopt (at least in this small 
sample). 

Of those firms that did not indicate significant IS cost pressure, the next question is 
whether they had a strong quality focus in place. One firm did not. As a result, they had 
the fewest measures in the sample. It would seem then that either cost or quality must be 
strong to have a fully developed performance measurement program. 

Of the firms that had little cost pressure and a strong quality focus, one was just now 
shifting its focus from availability and response measures (i.e., machine-oriented measures) 
to defect reduction and customer satisfaction (i.e., customer-oriented measures). This shift is 
the beginning of a thoroughgoing change in measurement strategy noticed in the last two 
firms. 


Differences also remain between the two firms that have pursued a quality 
measurement program after restructuring. The cultures of the two firms are very different - 
one emphasizes individuality, autonomy, and decentralized activity. Their performance 
measurement system "grew up" over the years without much central direction. The result is 
a measurement system which has become highly decentralized. Each team develops and 
maintains its own measurement system. These systems are not secret. In fact, they are 
plastered over every wall in the building. But they are not centrally rationalized or reported. 
Customer measures remain good (even better than before) so management is content that the 
decentralized system is working. 

The other firm is still quite centralized. While IS has changed much in the last few 
years, they still believe that their most effective strategy is to maintain a strong leadership 
position in IS for their business units. Their goal is to be valued business partners rather 
than merely a supplier of IS products and services. And their performance system mirrors 
that goal. Its key element is the satisfaction of the executive customer in each business unit. 
Those views are actively solicited and used in strategic and tactical planning. 

This framework is extremely preliminary. It represents more a direction for future 
research than an established theory. In the meantime, however, the framework might direct 
attention to the fact that the systems described are a function of the business environment and 
the stakeholder goals in each firm. 
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